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ABSTRACT  (U) 

(U)  Detection  of  potential  threats  that  are  camouflaged  or 
concealed  is  important  not  only  for  military  acquisition  problems 
but,  also  for  crowd  surveillance  as  well  as  tactical  use  such  as  on 
border  patrols.  Imaging  and  display  technologies  that  take 
advantage  of  photo-simulation  and  sensor  fusion  are  discussed  in 
this  paper.  A  comparison  of  the  detection  rates  of  visible,  infrared 
(IR)  and  sensor-fused  imagery  of  scenes  that  contain  a  Rocket 
Propelled  Grenade  (RPG)  were  made.  Image  fusion  was  achieved 
using  a  Gaussian  Laplacian  pyramidal  approach  with  wavelets  for 
edge  enhancement.  Three  types  of  images  were  also  compared  in 
terms  of  better  detection  of  concealed  weapons. 

Objectives  (UJ 

(U)  Objective  of  this  experiment  was  to  evaluate  the  detection  rate  of  three  different  types  of 
imaging  techniques.  Also  to  apply  entropy,  clutter  and  textured  clutter  metrics  to  the  image  and 
find  a  relationship  to  the  detection  rate.  A  linear  model  using  SPSS  was  also  achieved. 

1 L )  Image  preparation  and  subject  recruitment 

(U)  Images  were  taken  with  a  visual  and  IR  camera.  For  visual  images,  the  Sony  Digital 
camcorder  (DCR-TRV  240)  was  used.  An  OMEGA  (Indigo  Systems  Corporation,  7.5  -  13.5 
pm,  164H  x  128V,  uncooled  micro  bolometer  sensor  array)  system  was  used  which  was 
connected  to  the  Sony  camera  for  image  capture.  Images  were  initially  recorded  on  Hi8  digital 
tapes.  Each  image  was  digitized  with  the  Fast-studio  software.  Per  each  shooter  images,  the 
individual  image  was  sized  and  cropped  to  match  the  location  of  the  target  (shooter)  on  pixel  by 
pixel  bases.  Fused  images  were  prepared  using  the  Fusetool  application  in  Matlab  based 
program.  30  civilian  subjects  were  randomly  selected  from  the  employee  population  of 
TARDEC.  Each  subject  took  the  RPG  detection  test,  and  ranking  the  quality  of  images  test. 

(Uf  Experimental  procedure 
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(U)  Test  No.  1 :  RPG  detection  test. 

(U)  The  test  consisted  of  192  images  in  a  random  order  (4  shooters  x  2  shots  x  3  image  type  x 
2  target/no  target  x  4  image  condition).  Four  different  shooters  posed  for  the  RPG  fire.  Images 
were  taken  before  the  RPG  fire  and  after  the  RPG  fire  (consist  of  eight  image  sets).  There  are 
three  image  types  (IR,  visual,  and  fused  images).  For  the  original  IR  and  visual  image  sets,  no 
target  images  were  made  with  Photoshop  utility  then  no  target  fused  images  were  made.  The 
original  image  set  was  considered  as  clear  condition.  To  add  the  difficulty  of  the  detection,  noise 
was  added  to  the  clear  condition  image  sets.  Also  for  similar  manor,  contrast  was  adjusted  and 
the  brightness  was  adjusted. 

(U)  The  test  was  done  on  a  flat  panel  monitor.  Each  image  was  shown  for  1  second  and  then 
random  noise  pattern  was  shown  with  question.  The  test  subject  was  asked  to  click  “Yes”  or 
“No”  button,  depending  on  whether  he  could  see  the  RPG  in  the  picture  or  not.  After  clicking 
the  answer  it  is  asking  question  whether  the  subject  is  ready  to  view  the  next  image.  After  all 
192  images  were  shown  randomly  the  subject  was  requested  to  enter  some  comments.  “Yes”  or 
“No”  answer  was  recorded  per  image  and  response  time  was  also  recorded  (time  from  the  “Yes” 
or  “No”  button  shows  up  to  the  actual  mouse  clicking  on  the  answer)... 


Figure  1.  j  U)  Visual  images  of  shooterl,  2,  3,  and  4  respectively 


Figure  2.  (U)  IR  images  of  shooterl,  shooter  2,  shooter  3,  and  shooter  4  respectively 


(U)  Test  No.  2:  Visual  preference  of  three  types  of  images:  visual.  IR  and  fused. 

(U)  30  of  same  subjects  who  took  the  test  No.l  were  also  tested  for  the  ranking  quality  of  the 
image  test.  Image  sets  were  consist  with  32  sets  of  three  combined  images  (visual,  IR  and  fused 
image  with  RPG  in  order).  Image  sets  consisted  of  four  shooters  with  before  and  after  the  shot 
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with  all  image  conditions.  Only  images  with  RPG  were  used  in  this  test.  The  question  that  was 
shown  in  the  monitor  was  “Which  image  is  better  in  terms  of  detecting  the  RPG?”  They  were 
asked  to  rank  images  in  the  set  by  clicking  the  preferred  image  area.  The  response  was  marked 
as  the  image  number  (IR  =  1,  visual  =  2,  fused  =  3)  in  order  in  the  response  file.  Figure  3  shows 
the  example  sets  for  the  shooter  1  before  the  shot  (A  and  B),  and  after  the  shot  (C  and  D). 


(C)  (D) 

Figure  3.  (U)  Shooter  1  withRPG  combined  image  of  visual.  IR.  and  fused 

(A)  clear  condition,  (B)  noise  added,  f  C  >  brightness  adjusted,  f D  >  contrast  adjusted 


(U1  Summary  of  Test  Data. 

(U)  Test  No  1. 


(U)  The  probability  of  detection  (PD)  of  the  RPG  was  calculated  as  a  ratio  of  correct  answers 
to  the  total  number  of  target  pictures  (images  with  people  with  RPG,  96  images  total).  Detection 
rate  was  highest  for  the  IR  image  due  to  the  high  contrast  level  between  the  RPG  and  the 
background.  For  the  visual  image,  the  contrast  ratio  between  the  RPG  and  the  tree  background 
was  very  low;  it  gave  the  lowest  detection  rate  over  the  three  image  types.  Detection  rate  of  the 
noise  added  image  was  significantly  low  compared  to  the  other  effects.  But  the  rate  of  detection 
rate  drop  of  the  fused  image  was  relatively  slow  compare  to  the  IR  and  visual  images. 


Probability  of  Detection  Over  SensorType 


IR  Visual  Fused 

sensor  type 
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Detection  comparison  with  image  quality 


Figure  4.  (U)  Comparison  of  the  probability  of  detection  over  sensor  types 
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(U)  Also  entropy,  clutter,  and  textured  clutter  metric  were  calculated  over  individual  images. 
Figure  5  is  the  graph  of  each  entity  averaged  over  four  shooters.  The  entropy  value  of  different 
types  of  sensor  behaved  similarly  over  each  image  conditions.  Both  clutter  and  textured  clutter 
behaved  similarly  for  the  image  conditions  which  giving  the  low  value  for  the  contrast  adjusted 
image.  Also  the  visual  image  had  the  highest  textured  clutter  and  clutter  value  for  the  same 
image  conditions  for  both  textured  clutter  and  clutter  metrics. 


Figure  5.  (U)  Entropy,  textured  duller,  duller  averaged  over  shooters 


(U)  The  entropy  of  an  image  is  a  measure  or  the  information  content,  in  terms  of  gray  scale 
levels  and  is  also  related  to  the  texture  if  the  image.  The  maximum  value  the  entropy  metric  can 
take  on  is  eight  and  the  minimum  is  zero.  The  equation  used  for  the  calculation  of  the  entropy 
(ref.l)  is  shown  below. 


# =i>(s)iogp(g) 


g=0 


Where  P(g)  is  the  probability  of  a  gray  value  g,  and  the  range  of  g  is  [0,  1,  2, . ,  L-l]. 

(U)  The  clutter  metric  used  in  this  paper  is  a  metric  defined  by  Schmeider-Weathersby  (SW) 
(ref.  2).  The  SW  clutter  metric  is  written  as  following. 


Clutter  = 


Where  a,  is  the  variance  of  the  zth  cell  and  N  is  the  number  of  cells.  The  SW  clutter  metric 
divides  the  image  into  a  number  of  cells,  each  of  which  are  scaled  to  represent  twice  the  longest 
dimension  of  the  target,  and  then  the  variance  of  each  cell  is  divided  by  the  total  number  of  cells. 
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(U)  The  textured  clutter  metric  was  modified  from  the  clutter  metric.  The  standard  clutter 
SW  definition  of  clutter  was  modified  to  include  image  texture  by  calculating  the  gray  scale 
texture  for  each  cell  and  then  using  that  texture  in  place  of  the  variance  (ref.  3).  The  textured 
clutter  equation  is  as  following: 


Clutter  = 


WXrJ) 

N 


Where 

TUXr,0)  =  YLna,b\jXr,O) 

G  =  0  6=0 


Where  a  and  b  are  any  two  gray-scale  values  at  any  pixel  (j,k)  in  the  image  matrix,  r  is  the  step 
size,  and  q  is  the  direction  of  the  steps  (q  equals  0  means  stepping  horizontally).  P  is  the  fraction 
of  the  total  number  of  steps  whose  end  points  are  the  gray-scale  values  a  and  b,  respectively. 
The  mean  texture  is  defined  as  following: 


MTUXr)  =  -l~Yjnj,k,r,e) 

6 

(U)  Figure  6  below  shows  a  good  correlation  of  entropy  of  image  with  detection,  as  well  as 
good  correlation  of  clutter  and  textured  clutter  with  detection.  As  we  can  see  from  the  table  in 
the  right  hand  side,  the  correlation  coefficients  are  close  to  1 .  In  this  test,  textured  clutter  matrix 
gave  the  best  correlation  to  the  detection  rate. 


Average  (Shooter  Vs.  Metrics) 


Entropy  Clutter  Textured  %  DET 
Clutter 


Shooter 

Entropy 

Clutter 

Textured  Clutter 

%  DET 

Shooter  1 

6.4 

22.1 

31.6 

49.9 

Shooter  2 

6.7 

33.0 

46,6 

80.8 

Shooter  3 

6.7 

32.7 

46.8 

91.5 

Shooter  4 

6.2 

21.4 

30.4 

46.5 

Correlation  value 

Entropy  vs.  %  Detection 

0.96157 

Clutter  vs  %  Detection 

0.97709 

Textured  Clutter  vs  %  Detection 

0.98249 

Figure  6.  (U)  shooter  vs.  metrics  comparison 


(U)  We  wish  to  investigate  whether  there  are  differences  in  sensor  type  effect  the  ability  to 
detect  threats.  Two  factors  that  will  be  considered  will  be  sensor  type  shooter  profile.  An 
experiment  was  designed  using  three  levels  of  sensor  and  four  levels  of  shooter.  The  level  of 
effect  was  held  at  four,  time  at  one,  and  threat  at  one.  The  response  variable  is  whether  the 

subject  correctly  identified  a  shooter  with  an  RPG.  A  rank  transformation  was  applied  to  the 
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response  variable.  This  will  be  our  dependent  variable.  Once  a  subject  is  chosen,  the  order  in 
which  the  twelve  treatment  combinations  are  randomly  determined.  Furthermore,  subjects  differ 
in  their  skill  and  ability  to  detect  threats.  Thus,  we  shall  use  subjects  as  blocks.  Now,  we  have  a  3 
X  4  factorial  experiment  in  a  randomized  complete  block. 

(U)  The  complete  analysis  of  variance  for  this  experiment  is  summarized  in  Table  1.  The  first 
column  of  the  table  labeled  Source  is  the  effect  or  factor(s)  in  the  model.  The  second  column 
shows  the  type  III  sum  of  squares  (ref.4)  which  is  often  used  for  an  unbalanced  model  with  no 
missing  cells.  The  third  column  labeled  ”df’,  shows  the  degree  of  freedom  for  each  sum  of 
squares.  The  Mean  Square  column  shows  the  mean  square  of  each  effect.  This  is  obtained  by 
divide  the  sum  of  squares  of  each  effect  by  the  degrees  f  freedom  for  each  effect.  The  fifth 
column  s  the  F  statistics  and  shows  the  F  statistic  for  each  effect.  The  F  statistic  is  obtained  by 
dividing  the  mean  square  for  each  effect  by  the  mean  square  error  term  listed  at  the  bottom  of  the 
Mean  Square  column.  Column  six,  labeled  “sig,”  is  the  P  value  of  the  F  statistic  for  each  effect. 
The  P  value  is  the  probability  that  the  test  statistic  will  take  on  a  value  that  is  at  least  as  high  as 
the  observed  value  of  the  statistic  when  the  null  hypothesis  is  true.  Alternatively,  the  P  value  is 
the  smallest  level  of  significance,  or  a-value,  that  would  lead  to  rejection  of  the  null  hypothesis. 
The  smaller  the  P  value  is  the  greater  the  importance  of  the  effect.  Both  sensor  and  shooter  are 
significant.  The  interaction  of  these  two  factors  is  also  significant. 


Tests  of  Between-Subjects  Effects 


Dependant  Varaaple  RANK  of  RESPONSE 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

5-9 

Noncent 

Parameter 

Observed 

Power* 

Corrected  Mpdel 

1791720  000& 

40 

44793  000 

14  181 

000 

567  225 

1  000 

intercept 

11728800  0 

1 

11723890  00 

3713  147 

000 

3713  147 

1.000 

SUBJECT 

U0360Q0CI 

29 

§219.310 

1  969 

003 

57  099 

998 

SENSOR 

470340  000 

2 

235170  000 

74  450 

000 

146  901 

1  000 

SHOOTER 

807120  000 

3 

269040.000 

86.173 

.000 

255.510 

1.000 

SENSOR  *  SHOOTER 

333900.000 

6 

55650,000 

17618 

000 

105  707 

1  DOG 

Error 

1007640  D00 

319 

3153.746 

Total 

145282500 

360 

Corrected  Totei 

2799300.000 

35& 

a  Computed  using  alpha  =  .05 

b.  R  Squared  =  .640  (Adjusted  R  Squared  =  .595) 


Table  I.  <U)  ANOVA  of  test  factors 


(U)  When  the  interaction  is  significant,  comparisons  between  the  means  of  the  factor  sensor 
may  be  obscured  by  the  interaction.  One  approach  to  this  situation  is  to  fix  the  factor  shooter  at  a 
specific  level  and  apply  a  multiple  comparison  test  to  the  mean  of  the  factor  sensor  at  that  level. 
The  results  of  this  analysis  are  summarized  in  Table  2.  The  starred  values  indicate  pairs  of  means 
that  are  significantly  different.  For  example,  this  analysis  indicates  when  SHOOTER  =  2,  there 
is  a  statistically  significant  difference  between  SENSOR  =  3  (fused  image)  and  SENSOR  =  1 
(visual  image),  and  there  is  not  a  statistically  significant  difference  between  SENSOR  =  3  and 
SENSOR  =  2  (IR  image). 
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Pairwise  Comparisons 


Dependent  Variable;  RANK  of  RESPONSE 


SHOOTER 

II)  SENSOR  (J)  SENSOR 

Mean 

Difference 

Std,  Error 

Sig,a 

95%  Confidence  interval  for 
Difference3 

Lower  Bound 

Upper  Bound 

1 

1 

2 

-90.000' 

14.511 

000 

-118.550 

-61.450 

3 

-30,000' 

14.511 

.040 

-58.550 

-1.450 

2 

1 

90,000’ 

14.51 1 

.000 

61.450 

118.550 

3 

60.000* 

14.511 

.000 

31.450 

88.550 

3 

1 

30.000* 

14.511 

.040 

1.450 

58.550 

2 

-60.000* 

14.511 

,000 

-88,550 

-31.450 

2 

1 

2 

*138.000* 

14.511 

,000 

-166.550 

-109.450 

3 

*138.000* 

14.511 

.000 

-166.550 

-109.450 

2 

1 

138.000* 

14,511 

.000 

109.450 

166.550 

3 

-7.105E-14 

14.511 

1.000 

-28,550 

28.550 

3 

1 

138.000* 

14.511 

.000 

109,450 

166.550 

2 

7.105E-14 

14.511 

1.000 

-28,550 

28.550 

3 

1 

2 

6.000 

14,511 

.680 

-22.550 

34.550 

3 

6.000 

14.511 

.680 

-22.550 

34.550 

2 

1 

-6.000 

14,511 

.680 

-34.550 

22.550 

3 

-9.948E-14 

14,511 

1.000 

-28.550 

28.550 

3 

1 

-6.000 

14.511 

.680 

-34.550 

22.550 

2 

9  948E-14 

14.511 

1.000 

-28.550 

28.550 

4 

1 

2 

-132000* 

14,511 

.000 

-160.550 

-103.450 

3 

-24.000 

14.511 

.099 

-52.550 

4.550 

2 

1 

132.000* 

14.511 

.000 

103.450 

160.550 

3 

108.000* 

14.511 

.000 

79.450 

136,550 

3 

1 

24.000 

14.511 

.099 

-4.550 

52.550 

2 

-108.000* 

14.511 

.000 

-136.550 

-79.450 

Based  on  estimated  marginal  means 

*•  The  mean  difference  is  significant  at  the  .05  level. 

a*  Adjustment  for  multiple  comparisons:  Least  Significant  Difference  (equivalent  to  no  adjustments). 


Table  2.  (U)  Pair-wise  comparisons 
(U)  The  linear  model  for  this  experiment  is: 

y(i  j,k)  =  n  +  y(i)  +  PC')  +  (Tp)OJ  )  +  5(k)  +  e(i,j,k) , 

i  =  1,2,3  (visual,  IR,  fused  image  respectively) 
j  =  1, 2,3,4  (shooterl,  shooter2,  shooter3,  shooter4) 
k  =  1,2, . ,30  (subjectl,  subject2,  ...subject30) 

where  x(i)  represents  the  sensor  effect,  P(j)  represents  the  shooter  effect,  (x,P)(i,j)  is  the 
interaction,  8(k)  is  the  block  effect,  and  s(i  j,k)are  independent  and  identically  distributed  normal 
error  term. 

(U)  The  normal  probability  plot  of  the  residuals  is  shown  in  Figure  7.  There  is  no  severe 
indication  of  non-normality.  The  plots  of  the  residuals  versus  each  factor  and  block  respectively 
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give  no  indication  of  inequality  of  variance.  These  plots  are  not  included  in  the  paper.  Thus,  we 
have  checked  the  adequacy  of  the  assumed  model. 


Normal  R-R  Riot  of  Standardized  Residi 


Oji  20  OO  7S  1.00 


Obs«rv«d  Cum  Prob 

Figure  7.  (ID  The  normal  probability  plot  of  the  residuals. 


(U) _ Test  No  2 

(U)  The  second  test  was  performed  to  find  out  the  subject  preference  over  the  quality  of  the 
image  set.  By  presenting  three  sensor  images  on  the  screen  at  the  same  time,  subjects  had 
chances  to  compare  three  sensor  images  directly.  Then  they  ranked  the  image  set  by  easier  to 
detect  to  harder  to  detect  the  RPG.  Figure  8  shows  the  result  of  the  first  choice  of  all  the 
subjects.  IR  image  was  the  first  choice  that  subjects  to  look  at  to  detect  the  RPG.  The  test 
subjects  also  commented  on  the  fact  that  they  preferred  the  fused  image,  because  it  had  more 
details. 
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Figure  8.  (Uf  Image  quality  comparison 


(U)  Conclusion 

(U)  The  statistical  results  of  the  RPG  test  were  analyzed.  The  detection  rate  of  the  RPG  in 
the  image  was  analyzed  versus  the  sensor  type.  Subjects  had  the  highest  detection  rate  of  the 
RPG  using  the  IR  sensor,  and  the  lowest  detection  rate  was  with  the  visual  image.  In  this 
particular  image  set,  the  contrast  between  the  RPG  and  the  background  was  critical  factor  of  the 
detection  rate  difference.  Even  though  the  IR  image  has  the  higher  detection  rate  for  the  second 
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test,  subjects  commented  on  the  fact  that  they  preferred  to  look  at  the  fused  image,  because  it  had 
more  details.  One  of  the  objectives  was  to  see  the  relationship  between  the  probability  of 
detection  and  three  metrics.  The  overall  entropy  value  was  not  significantly  changed  over  either 
the  sensor  type  or  the  shooter  type.  The  clutter  and  textured  clutter  metrics  traces  better  to  the 
detection  rate  (Figure  6.).  The  highest  correlation  was  between  the  textured  clutter  and  the 
probability  of  detection.  The  initial  data  was  taken  with  stereo  system.  The  authors  intend  in  the 
future  to  compare  the  detection  rate  of  three  different  sensor  images  by  using  the  3D  image 
presentation. 
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